fix(frontend): send attached files from the expert side-panel chat and show the message at once (SECRT-2956 1/3) [1/3] - #15582

Dev moved the upload-and-send logic into helpers/sendWithAttachments.ts
(#15582), so the reply timing now rides on OutgoingMessage there instead
of an extra dispatch argument. Dev's new stable-onsend test (#15679)
expects the send's reply-timing start as its fifth argument.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…resets, a refused first send and partial codecov verdicts (Significant-Gravitas#15759)

This PR removes the causes of seven recurring e2e and CI flakes. Each
backend service exempts its startup heap from full garbage collections,
and the scheduler and copilot executor load the block registry at
startup. RPC servers and the Docker frontend keep idle connections
longer than their clients do. A chat message sent while the page is
still loading its AI connections waits for them instead of being
refused. Codecov posts a backend or frontend verdict only once every
shard has uploaded, and the e2e stack's Redis cluster is created only
once every shard answers.

### Why / What / How

**Why:** each of these failed tests or blocked merges for reasons
unrelated to the change under test. On 19 Oct, 6 of 17 Full-stack runs
failed between 22:49Z and 23:26Z, and Significant-Gravitas#15697 was ejected from the merge
queue. The e2e stall is tracked internally as SECRT-2966.

Resolves Significant-Gravitas#15754
Resolves Significant-Gravitas#15700
Fixes AUTOGPT-SERVER-A6D

Before and after, per class. "Before" is 80 Full-stack runs on dev,
merge-queue and PR heads on 18–19 Oct. "After" is 8 runs dispatched on
`336ef7cadf`, every fix here except the frontend send fix, whose
before-rate (2 in 80) is too low to show in 8 runs; its test fails
without it.

| Class | Before | After |
| ------------------------------------------------------------ |
----------------------------------------------- |
---------------------------------------------------------------------------------------------------
|
| Failed e2e jobs | smoke 7 of 80, journeys 10 of 80 | smoke 0 of 8,
journeys 0 of 8 |
| REST event-loop step of 2 s or more, per smoke job | 31 of 80 (longest
14.87 s) | 0 of 8 (longest 0.37 s) |
| Copilot executor loads the block registry during turns | 16 of 16
journeys jobs sampled | 0 of 8 |
| RPC `ReadError` or "Failed to process" in the REST log | 24 of 80
journeys jobs (62 lines) | 0 of 8 journeys jobs, 0 of 8 smoke |
| `socket hang up` on a reused connection | 2 tests (runs 39337657617,
39383564514) | 0 |
| Stack never starts: `redis-0` unhealthy | 2 of 426 e2e jobs since 18
Oct (both on 20 Oct) | reproduced locally; 0 in the runs below |
| Final backend codecov verdict posted before 12 of 12 uploads | 6 of 12
commits | 0 of 4 heads from part of the shards; 1 of 4 from the parent's
carried-forward coverage (see below) |

1. **Full garbage collections froze services for seconds.** A full
(generation 2) collection walks every tracked object while holding the
GIL, so every thread in the process stops until it finishes. A service
that has loaded the block registry holds 1.6–2.2M tracked objects,
nearly all long-lived startup state (block classes and their schemas).
Locally one full collection over that heap takes 559–776 ms, which is
the pause a production process pays each time; this is estimated from a
local measurement, and Sentry and the production logs were not checked
for it. On the e2e runner the heap is partly swapped out, so the
collector pages it back in while every thread waits. The REST server's
event loop then froze for 4–15 s, every `/api/proxy` call made meanwhile
waited as long, and a test waiting on the UI timed out: the Library
`:390`, `:497` and `:541` failures, and `publish-happy-path.spec.ts:9`,
whose search navigation waited on six `/api/proxy` calls the frozen
server held for 5.6 s.
2. **The copilot executor's first turns were slow.** It loaded the block
registry lazily, inside the first turns that needed it, and the heap
that load grew triggered a full collection: one took 6,146 ms during the
journeys' first turns (run 39387674137). The follow-up and reconnect
journeys then missed their 15 s wait for the answer to start streaming,
the failure behind 6 of the 13 failed journey tests.
3. **RPC calls failed with `httpx.ReadError`.** Every `AppService`'s
uvicorn closed an idle keep-alive connection after its default 5 s,
while the RPC clients kept it pooled for 30 s. A call written into a
connection as the service closes it is reset. In the journeys, the REST
server's call to the scheduler failed that way and the route answered
500 "Failed to process … /schedules": 4 of the 13 failed journey tests,
4 of 5 runs on Significant-Gravitas#15616, and the Significant-Gravitas#15697 ejection. The scheduler made it
likely: it loaded every block on its first new schedule, 12 s on the
runner, and froze for 6–7 s of it. In production this is
AUTOGPT-SERVER-A6D: the executor's `update_graph_execution_stats` call
to the database manager failed with `ReadError` 16 times on 9 Oct,
during one user's 20,953-node run, and its retry gave up. Idling alone
is harmless, because httpcore drops a connection the service has already
closed. The failure needs the close and the call to coincide, and a
freeze makes every pooled connection that receives a call meanwhile hit
the boundary at once. Reproduced against a test `AppService` in a child
process, called through the real `get_service_client`:

| Trial | Calls | Before: ReadError | After: ReadError |
| ------------------------------------------------------------------- |
----- | ----------------- | ---------------- |
| idle 6, 10 or 20 s, no stall | 60 | 0 | 0 |
| reuse within ±50 ms of the 5 s keep-alive | 102 | 11 | 0 |
| the caller's loop blocked 0.4–1.5 s just before the boundary | 60 | 4
| 0 |
| the service's loop blocked 7 s inside a request | 10 | 0 | 0 |
| the service's GIL held 2–6 s by another thread, across the boundary |
45 | 42 | 0 |

4. **The Docker frontend reset reused connections.** Node closes an idle
keep-alive socket after 5 s plus a 1 s buffer. Playwright's request
client keeps pooled sockets with no expiry of its own and does not
retry, so a test request on a socket idle for about 6 s met the server's
FIN and failed with `socket hang up`: after 5.99 s idle in
`copilot-build-agent-journey.spec.ts:30` (run 39337657617), and after
5.93 s in `copilot-markdown-happy-path.spec.ts:38` (run 39383564514).
5. **A first message sent during page load was refused.** Starting a
chat refused a send made before the AI connection list had loaded: "AI
connections are still loading. Wait a moment and try again". The
composer then put the text back. On a freshly loaded page that list took
1.8 s on the runner, and `copilot-build-agent-journey.spec.ts:30`
pressed Enter about 100 ms after that request finished, before the page
had rendered with the list. The message was never sent, and the test
waited 60 s for a stream (runs 39339438495 and 39337526897: 2 of the 13
failed journey tests). A user who sends that early gets the same refusal
and has to send again.
6. **The e2e stack sometimes never started.** `redis-init` creates the
Redis cluster once, as soon as the three shard containers have started,
but a started shard can take a second to accept connections. In runs
39396005336 and 39397892921 it reached `redis-0` 0.4–0.6 s too early
("Could not connect to Redis at redis-0:17000: Connection refused") and
exited. `redis-0`'s healthcheck needs `cluster_state:ok`, so it went
unhealthy 5 min later and the job failed before any test ran.
7. **Codecov posted verdicts from part of the shards.** It recomputed
its statuses after each upload. On 6 of 12 recent commits, its last
`codecov/project/platform-backend` verdict came from 8–11 of the 12
backend shard uploads, and nothing was re-posted after the rest arrived.
`Check PR Status` fails on a red one, which only a re-run clears.

**What:**

- `freeze_startup_heap()` runs once a service has started. It runs at
the end of the REST server's lifespan, after every `AppService`'s
lifespan (scheduler, database manager, notifications, platform linking,
the copilot chat bridge, the swap service, the batch executor), and
after the graph executor's and the copilot executor's block preload.
Later full collections then walk only what was allocated after startup.
- The scheduler and the copilot executor load the block registry before
they serve or take turns, so the freeze covers the blocks too.
- RPC services keep an idle connection for 60 s, and clients retire
theirs at 30 s.
- The Docker frontend keeps idle connections for 65 s.
- Starting a chat waits for the AI connection list when it is still
loading, and refuses only if the list never arrives.
- `redis-init` waits up to 30 s for each shard to answer `PING` before
creating the cluster, as backend CI's cluster setup already does.
- Codecov waits for 12 `platform-backend` uploads and 4
`platform-frontend` uploads before posting those flags' statuses.

**How:** the helper calls `gc.collect()` and then `gc.freeze()`, and
logs how many objects it froze and how long that took. It collects
first, so garbage still in the heap is freed rather than frozen in place
for good. For `AppService`, `build_fastapi_app` wraps the service's
lifespan and freezes after the subclass's whole startup has run.
`AppService` passes `timeout_keep_alive` to uvicorn, and both RPC
clients derive `keepalive_expiry` from it as half of it. The 30 s margin
also covers a caller whose own event loop stalls between taking a
connection from the pool and writing to it. Next's standalone server
reads `KEEP_ALIVE_TIMEOUT`. `createSession` awaits `refetch({
cancelRefetch: false })` on the transports query, which joins the
request already in flight rather than starting a second one. Codecov's
per-flag `after_n_builds` counts each flag's own uploads, so a PR that
runs only the frontend suite doesn't wait for backend shards.

Borderline decisions:

- Every `AppService` freezes its heap after startup, including services
that never load blocks, because the call sits in the base class's
lifespan wrapper (`_serving_lifespan`). The freeze costs one collection
at startup: 0.8–3.0 s on the runner.
- The websocket server has no freeze, because it doesn't load the block
registry.
- The RPC service's keep-alive was raised, rather than the client's
expiry lowered. An expiry under 5 s would reconnect on nearly every
scheduler call, and would leave under a second of margin against a stall
in the caller.
- There is no unit test for the scheduler's preload. The only assertion
available would pin a call inside a heavily mocked `run_service`. Its
startup log line ("Loaded N blocks in …") shows it instead.
- The frontend's keep-alive moves the race with a client that never
expires its sockets from about 6 s of idleness to about 66 s; it cannot
remove it.
- A send made while the connection list loads waits as long as that
request takes, with no deadline of its own.
- When another flag uploads before the first backend shard, Codecov
counts the parent's carried-forward backend uploads toward the 12, and
posts `codecov/project/platform-backend` from them. On `c08de8480f` it
posted at the first frontend upload and was not re-posted after the 12th
backend upload, while `codecov/patch/platform-backend` waited for all
12. That verdict is the parent's coverage, so it cannot be a
partial-shard red, but it does not reflect this PR's backend coverage
either. On the other three heads checked, both backend statuses waited
for 12 of 12.
- A PR that runs no backend tests may get no `platform-backend` codecov
status, if Codecov does not count carried-forward uploads toward
`after_n_builds`; that was not tested here. It blocks nothing, because
`Check PR Status` is the only required check.

Known flakes this PR does not touch: the credits rate-limit clock, owned
by Significant-Gravitas#15452, and the ExpertChatDrawer attachments test, Significant-Gravitas#15582.

Made diagnosable, not fixed:
`e2e_redis_restart_test.py::test_subscriber_survives_shard_restart`
failed in 1 of 415 `data`-shard jobs (run 39391205056) with "port 27111
never opened within 60s" and nothing else to go on. Its timeout now
appends the cluster's `docker ps -a` and the shard's last 50 log lines.

### User-visible change

A message typed and sent while the chat page is still loading goes out
once the page has loaded its AI connections, instead of being refused
with "AI connections are still loading". If the connections never load,
the send is refused with "Could not check AI connections", as a failed
load already was.

### Changes 🏗️

- `backend/util/process.py`: `freeze_startup_heap()`.
- `backend/util/service.py`: `AppService.build_fastapi_app` wraps the
lifespan so the freeze runs after the subclass's startup.
`rpc_server_keepalive_seconds = 60` goes to uvicorn's
`timeout_keep_alive`, and `rpc_client_keepalive_expiry_seconds`, half of
it, replaces both hard-coded `keepalive_expiry=30.0`.
- `backend/api/rest_api.py`: the REST lifespan freezes after the catalog
load.
- `backend/executor/manager.py`: the graph executor freezes after its
block preload, before its worker threads start.
- `backend/executor/scheduler.py`: `run_service` loads the block
registry before serving, and logs how long that took.
- `backend/copilot/executor/manager.py`: the copilot executor loads the
block registry and freezes before it takes turns.
- `frontend/src/app/(platform)/copilot/useChatSession.ts`:
`createSession` waits for a connection list still loading.
- `docker-compose.platform.yml`: `KEEP_ALIVE_TIMEOUT: "65000"` on the
frontend, and `redis-init` waits for every shard's `PONG`.
- `codecov.yml`: `after_n_builds` on the `platform-backend` (12) and
`platform-frontend` (4) flags.
- `backend/data/e2e_redis_restart_test.py`: the port-wait timeout
reports the shard's container state and log.
- Tests:
- the helper freezes the heap, and collects garbage instead of freezing
it;
  - an `AppService` freezes after its own startup;
- both executors load the blocks and freeze before their threads start;
- a pooled RPC call made while the service's GIL is held past the
connection's idle time is answered rather than reset;
- a chat started while the connection list loads waits for it and is
created, and one whose list never arrives is refused.

### Agents and large language models used

- Claude Code with Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Unit tests for the helper and every call site, each shown failing
under the mutation it guards
- [x] `architecture_test.py`, `test_block.py` and the touched suites,
backend and frontend
- [x] Full-stack CI dispatched on this branch, failures and loop blocks
counted against the 80-run baseline
- [x] Reproduced the `ReadError` and the `socket hang up` races, and
re-ran them with the fixes

**Verified:** On the backend I ran these, all passing:

- `process_test.py`, `service_test.py` and `swap_service_test.py`;
- both executors' `manager_block_warmup_test.py`, plus `manager_test.py`
and `manager_cancel_consumer_test.py`;
- `entry_points_test.py`, `capability_tools_test.py` and
`architecture_test.py`;
- `test_block.py` (3,696 passed, 84 skipped).

Each new backend test went red under its mutation: dropping
`gc.freeze()` or `gc.collect()`, moving the freeze before the subclass's
startup, building the app from `self.lifespan`, and dropping either
executor's preload or freeze. The keep-alive test failed 5 of 5 times
with `httpx.ReadError` without the server keep-alive, and passes with
it. On the frontend, the copilot suite passes (314 files, 4,131 tests),
as do `tsc` and `eslint`. The two new `createSession` tests fail against
the old hook (2 failed, 9 passed) and pass with the fix. A Node client
reusing a socket after 5.990–6.005 s idle failed 4 of 4 times against
Node's default keep-alive, and succeeded 4 of 4 times with
`KEEP_ALIVE_TIMEOUT=65000`. Codecov's validator accepts `codecov.yml`.
With `redis-0` started 3 s late in a throwaway compose project, the old
`redis-init` exited 1 with "Connection refused" and no cluster formed.
The new one waited and formed it (`cluster_state:ok`), and it still
forms one in 2 s with no delay. The Full-stack measurements are in the
table above. Two more runs on `4eadfd936e`, which has the frontend send
fix, repeated the journeys three times, and both legs passed (the
build-agent journey 6 of 6; runs 39393594364 and 39393597786). Run ids
and logs are in the comments below.

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…hment tests (Significant-Gravitas#15792)

### Why / What / How

**Why:** "sends a workspace file picked mid-reply as the next turn" in
the ExpertChatDrawer attachments tests (added in Significant-Gravitas#15582) has failed
often in CI and the merge queue since the React 19 upgrade (Significant-Gravitas#15746). It
knocks unrelated PRs out of the queue. The failure is `Unable to find
role="button" and name /brief\.pdf/` in `pickWorkspaceFile()`, and
sometimes the test hits its 5 s timeout instead.

**What:** `pickWorkspaceFile()` now finds the picker dialog first and
looks for the file button inside it, not across the whole page.

**How (root cause):** This is not a logic race. The picker opens and the
file list request goes out; the list just hasn't rendered when findBy
gives up. The time goes into the query itself. A page-wide `*ByRole`
check reads the computed style of every element, and happy-dom
recomputes those styles after every DOM change. Measured locally, the
first page-wide check after the picker opens takes 100-200 ms, while the
same check scoped to the dialog takes about 15 ms. A full style pass
over the page takes 240-640 ms. These checks run on the main thread, so
they also hold back the mock response and the render the test is waiting
for. On a busy CI runner that uses up findBy's default 1 s. "sends a
workspace file picked in an existing thread" uses the same helper and
failed the same way locally.

### Changes 🏗️

- `attachments.test.tsx`: `pickWorkspaceFile()` gets the dialog "Use a
file from your workspace" and runs the `brief.pdf` and "Add 1 file"
queries inside it with `within()`. No timeouts were raised and no sleeps
were added. No product code changed.

### Agents and large language models used

- Claude Code — Claude Opus 5.5

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Ran the whole test file 112 times before the fix, 14 runs in
parallel on 14 cores (part of them with 8 extra CPU burners): **5
failures** (all at the `brief.pdf` step, in both tests that use the
helper, plus one 5 s timeout)
  - [x] Same 112 runs under the same load after the fix: **0 failures**
  - [x] Whole `ExpertChatDrawer` test folder passes (28 tests)
  - [x] `pnpm format`, `pnpm lint`, `pnpm types` pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>